[DO NOT MERGE] [Klaud Cold] experimental: profile MiniMax-M3 MI325X conc 4/8 — vllm#45639 (AITER AR + Gemma-RMS fusion)#1773
Conversation
… applied MI325X (gfx942) counterpart of #1770. Validate vllm-project/vllm#45639 ("[ROCm][M3] Enable AITER AR + Gemma-RMS fusion for MiniMax-M3") on MI325X before an image rebuild, by applying the PR diff in-place to the shipped minimax-m3 image. - patches/vllm-45639-aiter-ar-gemma-rms.diff: vendored PR diff. - minimaxm3arf_fp8_mi325x.sh: applies the diff (idempotent; HARD-FAILS if it neither applies nor is already applied), serves with VLLM_ROCM_USE_AITER=1 + --compilation-config (custom_ops -minimax_gemma_rms_norm, pass_config.fuse_allreduce_rms). BF16 KV (gfx942). Includes a PROFILE=1 --profiler-config gate so the same recipe serves the companion profiling PR. - amd-master.yaml minimaxm3arf-fp8-mi325x-vllm: model-prefix minimaxm3arf routes to the new recipe; conc 4 and 8, TP8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…39 patch MI325X profiling counterpart of #1771 (and of the #1772 MI325X sweep). Capture Perfetto torch traces for MiniMax-M3 MXFP8 on MI325X (TP8, conc 4 and 8) with vllm-project/vllm#45639 (AITER fused all-reduce + Gemma-RMSNorm) applied in-place. - Label-triggered profile.yml (add 'profile-enabled'); reads .github/profile-target.env. - profile-target.env -> minimaxm3arf-fp8-mi325x-vllm, conc 4 8. - The minimaxm3arf_fp8_mi325x.sh recipe (from #1772) already carries the --profiler-config PROFILE gate + the #45639 in-place patch + AITER fusion flags. Built on the #45639 MI325X recipe/config/patch (#1772); both are DO NOT MERGE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
functionstackx
added
the
profile-enabled
functionstackx
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
…sent) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit edfa135. Configure here.
| git rebase origin/master | ||
| if [ "$attempt" = 8 ]; then echo "push failed after ${attempt} attempts" >&2; exit 1; fi | ||
| sleep $(( (RANDOM % 6) + 2 )) | ||
| done |
There was a problem hiding this comment.
PR traces use merge SHA
Medium Severity
Label-triggered profiling checks out the PR head commit via needs.get-jobs.outputs.ref, but trace storage paths, commit messages, and Perfetto raw URLs still use GITHUB_SHA. On pull_request runs that SHA is GitHub’s synthetic merge commit, not the profiled head, so artifacts and relay links won’t align with the PR revision that was actually run.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit edfa135. Configure here.
1 participant
What
MI325X (gfx942) profiling counterpart of #1771. Capture Perfetto torch traces for MiniMax-M3 MXFP8 on MI325X (TP8, conc 4 and 8) with vllm-project/vllm#45639 (AITER fused all-reduce + Gemma-RMSNorm) applied in-place.
How
profile.yml: add theprofile-enabledlabel → it reads.github/profile-target.env, runs one profile job per(conc, tp), emits a Perfetto trace (artifact + relay link) per job.profile-target.env→CONFIG_KEY=minimaxm3arf-fp8-mi325x-vllm,CONC=4 8(TP8 → 2 jobs).minimaxm3arf_fp8_mi325x.shrecipe (from [DO NOT MERGE] [Klaud Cold] experimental: MiniMax-M3 MI325X conc 4/8 — apply vllm#45639 (AITER AR + Gemma-RMS fusion) #1772) already carries the--profiler-configPROFILE=1gate, the in-place #45639 patch (idempotent + hard-fail on drift), and the AITER fusion serve flags. BF16 KV (gfx942).Built on the #45639 MI325X recipe/config/patch from #1772 — both are DO NOT MERGE.
Validation
profile.ymlparses ✓;test-config+(conc,tp)filter → 2 jobs (TP8, conc 4 + 8, mi325x,minimaxm3arf_1k1k) ✓.To run: add the
profile-enabledlabel.🤖 Generated with Claude Code
Note
Medium Risk
Changes the shared Profile workflow (PR triggers, matrix fan-out, git push retry) and adds experimental in-container vLLM patching on hardware runners; scope is CI/benchmark-only and marked DO NOT MERGE.
Overview
Adds an experimental (DO NOT MERGE) path to capture Perfetto torch traces on MI325X for MiniMax-M3 MXFP8 with upstream vllm#45639 (AITER fused all-reduce + Gemma-RMSNorm) applied in-place before serve.
Label-triggered profiling:
profile.ymlnow runs on PRs with theprofile-enabledlabel, reads.github/profile-target.env(CONFIG_KEY,CONFIG_FILE,CONC), checks out the PR head SHA, and builds a matrix of one job per (concurrency, tp) instead of a single job. Profile jobs setSCENARIO_SUBDIR=fixed_seq_len/and use rebase/retry when parallel jobs push traces to the storage repo.Benchmark wiring: New
minimaxm3arf-fp8-mi325x-vllminamd-master.yaml(TP8, conc 4/8 smoke), recipeminimaxm3arf_fp8_mi325x.sh(idempotent vllm patch, AITER + compilation fusion flags,--profiler-configwhenPROFILE=1), and vendoredvllm-45639-aiter-ar-gemma-rms.diff.perf-changelog.yamldocuments the config.Reviewed by Cursor Bugbot for commit edfa135. Bugbot is set up for automated code reviews on this repo. Configure here.